Image processing apparatus, control method thereof, and storage  medium

ABSTRACT

An image processing apparatus comprises: an image generation unit configured to generate one or more reduced images based on an input image, the input image and the reduced images forming hierarchized images; a gain map generation unit configured to generate a gain map for each hierarchical level by applying a first tone characteristic to each of the hierarchized images and also further applying a second tone characteristic to at least one of the reduced images; and a composing unit configured to compose the created gain maps to generate a composed gain map, wherein the gain map generation unit applies, for each subject region, the first tone characteristic and the second tone characteristic to the at least one of the reduced images, and wherein the first tone characteristic and the second tone characteristic are mutually different tone characteristics.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, acontrol method thereof, and a storage medium.

Description of the Related Art

In recent years, a local tone mapping technique for improving thecontrast of an image by applying a gain defined for each luminancesignal is known.

In a case of performing tone processing that applies a gain defined foreach luminance signal, when the gain is applied to a range of a specificluminance signal to brighten a predetermined subject region, there arecases where other regions brighten and the contrast decreases. Forexample, in a scene where there is a person region 501 as illustrated inFIG. 5A, there is a range where the brightness of the person region 501and the brightness of a region 502 that is not a person region overlap,as illustrated in FIG. 5B. For this reason, in a case of applying astrong gain to brighten the person region, a region that is not a personregion in which the distribution of brightnesses overlaps also becomesbrighter, and the contrast of the non-person region decreases.

With respect to a problem where the brightness of a region that is not adesired subject region changes, a method of applying tonecharacteristics that differ per subject region may be considered.Japanese Patent Laid-Open No. 2009-272983 discloses a technique ofsetting a region in accordance with a level of a pixel, and applyingapproximately the same gain to the set region.

As with the conventional technique described above, in a case ofapplying tone characteristics that differ for each region (for example,for each subject), it is assumed that setting of a region(discrimination of a subject region) can be performed with highaccuracy. However, typically there are many cases where it is difficultto perform discrimination of a subject region under various imagecapturing conditions with high accuracy, and there are cases where adiscriminated region includes an incorrect detection. In other words, ina case where discrimination of a subject region cannot be performed withhigh accuracy (that is the discriminated region includes an incorrectdetection), if tone characteristics that differ per region are applied,then the brightness will change for an unintended region. For thisreason, even if there is an error in discrimination of a subject regionin a case of applying tone characteristic that differ for each region, atone correction technique that can reduce the influence of the error isdesired.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theaforementioned problems, and realizes a technique that can, even ifthere is an error in discrimination of a subject region in tonecorrection that applies tone characteristics that differ for eachsubject region, reduce the effect of the error.

In order to solve the aforementioned problems, one aspect of the presentinvention provides an image processing apparatus comprising: an imagegeneration unit configured to generate one or more reduced images thatare reduced in a stepwise fashion based on an input image, the inputimage and the reduced images forming hierarchized images; a gain mapgeneration unit configured to generate a gain map for each hierarchicallevel that represents a gain at each pixel position of an image, byapplying a first tone characteristic to each of the hierarchized imagesand also further applying a second tone characteristic to at least oneof the one or more reduced images; and a composing unit configured tocompose gain maps generated for each hierarchical level to generate acomposed gain map that represents gain applied to each pixel position ofthe input image, wherein the gain map generation unit applies, for eachsubject region, the first tone characteristic and the second tonecharacteristic to the at least one of the one or more reduced images,and wherein the first tone characteristic and the second tonecharacteristic are mutually different tone characteristics associatedwith a respective subject region included in the input image.

Another aspect of the present invention provides, an image processingapparatus, comprising: an image reduction unit configured to generateone or more reduced images by reducing an input image and generate animage group that includes the input image and the one or more reducedimages; and a gain map generation unit configured to respectivelygenerate a gain map that indicates a gain value at each position in animage by applying a predetermined tone characteristic to each image inthe image group, wherein the gain map generation unit generates the gainmap by applying, to the image out of the image group for which aresolution is relatively lowest, tone characteristics for which acharacteristic differs for each subject region included in the image; again map composing unit configured to generate a composed gain map bycomposing a plurality of gain maps generated based on each image of theimage group; and an image conversion unit configured to perform a toneconversion by using the composed gain map on the input image.

Still another aspect of the present invention provides, a control methodof an image processing apparatus, the method comprising: generating oneor more reduced images that are reduced in a stepwise fashion based onan input image, the input image and the reduced images forminghierarchized images; generating a gain map for each hierarchical levelthat represents a gain at each pixel position of an image, by applying afirst tone characteristic to each of the hierarchized images and alsofurther applying a second tone characteristic to at least one of the oneor more reduced images; and composing gain maps generated for eachhierarchical level to generate a composed gain map that represents gainapplied to each pixel position of the input image, wherein, for eachsubject region, the first tone characteristic and the second tonecharacteristic are applied to the at least one of the one or morereduced images, and wherein the first tone characteristic and the secondtone characteristic are mutually different tone characteristicsassociated with a respective subject region included in the input image.

According to yet another aspect of the present invention, an informationprocessing system comprises, a control method of an image processingapparatus, the method comprising: generating at least one or morereduced images by reducing an input image and generate an image groupthat includes the input image and the at least one or more reducedimages; respectively generating a gain map that indicates a gain valueat each position in an image by applying a predetermined a tonecharacteristic to each image in the image group, wherein the gain map isgenerated by applying, to an image out of the image group for which aresolution is relatively low, tone characteristics for which acharacteristic differs for each subject region included in the image;generating a composed gain map by composing a plurality of gain mapsgenerated based on each image of the image group; and performing a toneconversion by using the composed gain map on the input image.

According to still yet another aspect of the present invention, anon-transitory computer-readable storage medium storing a computerprogram for causing a computer to function as an image processingapparatus comprising: an image generation unit configured to generateone or more reduced images that are reduced in a stepwise fashion basedon an input image, the input image and the reduced images forminghierarchized images; a gain map generation unit configured to generate again map for each hierarchical level that represents a gain at eachpixel position of an image, by applying a first tone characteristic toeach of the hierarchized images and also further applying a second tonecharacteristic to at least one of the one or more reduced images; and acomposing unit configured to compose gain maps generated for eachhierarchical level to generate a composed gain map that represents gainapplied to each pixel position of the input image, wherein the gain mapgeneration unit applies, for each subject region, the first tonecharacteristic and the second tone characteristic to the at least one ofthe one or more reduced images, and wherein the first tonecharacteristic and the second tone characteristic are mutually differenttone characteristics associated with a respective subject regionincluded in the input image.

According to the present invention, it is possible to, even if there isan error in discrimination of a subject region in tone correction thatapplies tone characteristics that differ for each subject region, reducethe influence of the error.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram illustrating an example of a functionalconfiguration of an image processing unit according to a firstembodiment.

FIG. 2 is a flowchart for describing a sequence of operations for gainprocessing according to the first embodiment.

FIGS. 3A to 3D are views describing a tone characteristic, a gain map,and region discrimination result.

FIG. 4 is a view illustrating an example of a first tone characteristicand a second tone characteristic according to the first embodiment.

FIGS. 5A to 5B are views describing an example of a tone correction onan input image.

FIG. 6 is a block diagram illustrating an example of a functionalconfiguration of an image processing unit according to a secondembodiment.

FIG. 7 is a flowchart for describing a sequence of operations for gainprocessing according to the second embodiment.

FIG. 8 is a view illustrating an example of a first tone characteristic,a second tone characteristic, and a third tone characteristic accordingto the second embodiment.

FIGS. 9A to 9B are views describing an example of a tone correctionaccording to the second embodiment.

FIGS. 10A and 10B are views for giving an explanation regarding a methodof deciding a hierarchy based on a result of discriminating a subject.

FIG. 11 is a view for illustrating an example of a functionalconfiguration of a digital camera as an example of an image processingapparatus according to the first and second embodiments.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

Exemplary embodiments of the present invention will be explainedhereinafter in detail, with reference to the drawings. Explanation isgiven below of an example in which a digital camera that can performgain processing on an inputted image is used as an example of an imageprocessing apparatus. However, the present embodiment is not limited toa digital camera, and can also be applied to electronic devices that canperform gain processing on an inputted image. These devices may include,for example, a personal computer, a mobile telephone including a smartphone, a game device, a tablet terminal, a watch-type or glasses-typeinformation terminal, a medical device, a device for a monitoring systemor an in-vehicle system, or the like.

(Configuration of Digital Camera)

FIG. 11 illustrates an example of a functional configuration of adigital camera 100 as an example of an image processing apparatus. Notethat, one or more of the functional blocks shown in FIG. 11 may berealized by hardware such as an ASIC, a programmable logic array (PLA),or the like, and may be realized by a programmable processor such as aCPU, an MPU, or the like, executing software. Also, these may berealized by a combination of software and hardware. Accordingly, in thefollowing explanation, even in a case where different functional blocksare recited as the subject of operations, it is possible that this maybe realized by the same hardware as the subject.

An optical system 1101 includes a shutter apparatus, an apertureadjustment apparatus, and a lens group formed from a zoom lens and afocus lens. In accordance with an instruction by a control unit 1105,the optical system 1101 adjusts an amount of light incident on an imagecapturing unit 1102, or a focus position or a scaling factor of asubject image incident on an image capturing element included in theimage capturing unit 1102.

The image capturing unit 1102 includes an image capturing element thathas a configuration in which a plurality of pixels that each have aphotoelectric conversion element is arrayed two-dimensionally. The imagecapturing element performs a photoelectric conversion by each pixel ofan optical image of a subject that is formed by the optical system 1101,and successively outputs an analog signal. The image capturing elementmay be an image capturing element such as a CCD (Charge-Coupled Device)image sensor, a CMOS (Complementary Metal Oxide Semiconductor) imagesensor, or the like. An A/D conversion unit 1103 performs, in accordancewith an A/D conversion circuit, an analog/digital conversion of ananalog signal outputted from the image capturing unit 1102, and outputsa digital signal (image data) with units of pixels.

An image processing unit 1104 performs development processing such aspixel interpolating processing, and predetermined compression processingfor storing an image to a storage unit 1110. In addition, it performsgain processing according to the present embodiment which is separatelydescribed later, and generates an output image to which the gainprocessing has been applied. The image processing unit 1104 is notlimited to an image outputted from the A/D conversion unit 1103, and canalso perform similar image processing or decompression processing to animage read from the storage unit 1110. Note that the image processingunit 1104 may be realized by a dedicated image processing circuit, andfunctions of the image processing unit 1104 may be realized throughsoftware in accordance with the control unit 1105 executing a program.

The control unit 1105 includes a CPU or an MPU, a ROM, a RAM or the likefor example, and controls the entirety of the digital camera 100 byloading a program stored in the ROM into a work area of the RAM andexecuting it.

A console unit 1106 includes an operation member such as a touch panel,a dial, a button, or a switch, for example, and conveys an operation bya user to the control unit 1105. If part of the console unit 1106 is atouch panel, a captured or stored image may be displayed on the touchpanel in addition to displaying an operation menu thereon. The storageunit 1110 includes a storage medium such as a magnetic disk or a memorycard in which a semiconductor memory is mounted for example, and storesa captured image or reads out a stored image.

(Configuration of Image Processing Unit 1104)

FIG. 1 is a block diagram that illustrates an example of a functionalconfiguration of the image processing unit 1104 according to the presentembodiment. The image processing unit 1104 performs gain processing withrespect to an inputted image, and then outputs the image. Note that theimage data outputted from the A/D conversion unit 1103 is converted to aluminance signal from RGB data by a luminance value calculation unit(not shown), and an image formed by this luminance signal is inputted tothe image processing unit 1104. The image processing unit 1104 includesa subject region discrimination unit 103 for discriminating a subjectregion from an input image. It also includes a first reduced imagegenerating unit 101 and a second reduced image generating unit 102 forgenerating hierarchized images by gradually reducing the input image. Itfurther includes a first gain conversion unit 104 and a second gainconversion unit 105 for converting a signal of a hierarchized image to again signal. In addition, it includes a by-region gain map composingunit 106 that composes a plurality of gain maps (gain maps) thatcorrespond to the generated reduced images, in order to perform gain mapgeneration for each subject region. It further includes a hierarchicalgain map composing unit 107 that generates from the hierarchized gainmaps a gain map to finally apply to the input image, and a gainprocessing unit 108 for applying a calculated gain to the input imageand outputting an output image for which tone conversion processing hasbeen performed (the gain is changed). The operation of each of theseblocks is described below together with a series of operations that areindicated in FIG. 2.

(Series of Operations for Gain Processing)

Next, with reference to FIG. 2, explanation is given regarding a seriesof operations for gain processing. Note that gain processing is startedwhen captured image data is outputted from the A/D conversion unit 1103and a luminance signal based on the image data is inputted to the imageprocessing unit 1104.

In step S201, the subject region discrimination unit 103 discriminates ameaningful subject region from the input image. In the presentembodiment, a person region is discriminated for example. Fordiscrimination of the subject region, it is possible to use a typicalregion discrimination method that uses learning data or a feature amountof an image, as described in Japanese Patent Laid-Open No. 2006-39666for example.

For example, in a case of discriminating a person region in a scene asillustrated in FIG. 3C, a result of discriminating a subject region is amonochrome image as illustrated in FIG. 3D, for example. In the exampleof FIG. 3D, a region discriminated to be a person region is a whiteregion illustrated by a region 301, and conversely a region other than aperson region is a black region as with a region 302. In the presentembodiment, a range of signal values from a white region to a blackregion is outputted as a signal value from 255 to 0. The signal valueindicates a value from 0 to 100% of a reliability of the regiondiscrimination result, and for example 255 is assigned to a reliabilityof 100%, and 0 is assigned to a reliability of 0%. Note that, in thepresent embodiment, a region that is not a discriminated subject isreferred to as a background region.

In step S202, the first reduced image generating unit 101 generates afirst reduced image by performing reduction processing on the inputimage, and the second reduced image generating unit 102 generates asecond reduced image based on the first reduced image. The input image,the first reduced image, and the second reduced image form hierarchizedimages (an image group) for which image sizes (or a degree of reduction)are different in a stepwise fashion. In the example of the presentembodiment, the second reduced image is an image resulting fromsubjecting the first reduced image to further reduction processing. Notethat, regarding the method of the reduction processing, it is possibleto use a general method that is publicly known, such as reductionprocessing that uses a bi-linear method, for example.

In step S203, the first gain conversion unit 104 applies a first tonecharacteristic to the hierarchized images (the image group formed by theinput image, and the first reduced image and the second reduced imagethat are generated in step S202), and generates a gain map thatcorresponds to each image. The tone characteristic is a table thatindicates a relationship of gain to an input luminance signal, and, inthe present embodiment, indicates a gain table in which the abscissaindicates an input luminance signal and the ordinate indicates a gainsignal, as illustrated by FIG. 3A. A first tone characteristic is a tonecharacteristic associated with a background region. Details of the firsttone characteristic are described later. By applying the first tonecharacteristic to each of the hierarchized images, the first gainconversion unit 104 respectively generates a gain map formed by a gainsignal for each position (each pixel position) in an image asillustrated in FIG. 3B. Note that the gain map may be in a format suchas a table if it is something that represents a gain signal (a gainvalue) of each position (each pixel position) in an image.

In step S204, the second gain conversion unit 105 generates a gain mapby applying a second tone characteristic to an image of a lowest level(the second reduced image). A second tone characteristic is a tonecharacteristic associated with a person region. The second tonecharacteristic is described later.

In step S205, the by-region gain map composing unit 106 generates a gainmap for the lowest level by composing the two gain maps of the lowestlevel (having the size of the second reduced image) to which the firsttone characteristic and the second tone characteristic have beenrespectively applied. The by-region gain map composing unit 106 uses theregion discrimination result generated in step S201 as a compositioncoefficient that is used for the composing. The region discriminationresult is reduced to the size of the second reduced image. The by-regiongain map composing unit 106 calculates Gaincomp(x, y), the value of thegain map resulting from composing, in accordance with Equation 1.However, with respect to the coordinates (x, y) of the second reducedimage, let the value of the gain map generated in accordance with thefirst tone characteristic be Gain1(x, y), and the value of the gain mapgenerated in accordance with the second tone characteristic be Gain2(x,y). In addition, let the signal value of the region discriminationresult be area(x, y), and the maximum value that can be taken for thesignal value for region discrimination be area_MAX.

$\begin{matrix}{{{Gaincomp}\left( {x,y} \right)} = \frac{\begin{matrix}{{{\left( {{area\_ MAX} - {{area}\left( {x,y} \right)}} \right) \cdot {Gain}}\; 1\left( {x,y} \right)} +} \\{{{{area}\left( {x,y} \right)} \cdot {Gain}}\; 2\left( {x,y} \right)}\end{matrix}}{area\_ MAX}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

Note that, although the region discrimination result area(x, y) is usedunchanged in the example described above, a signal value for which aboundary of a discrimination result is made to not stand out by applyinga low-pass filter, a bilateral filter, or the like may be used for theregion discrimination result.

In step S206, the hierarchical gain map composing unit 107 generates afinal gain map (a composed gain map) by composing three gain maps: thegain maps generated from each of the input image and the first reducedimage in step S203, and the gain map generated from the second reducedimage in step S205. For the processing for composition of the gain maps,it is possible to use a publicly known method (for example, JapanesePatent Laid-Open No. 2014-154108) for performing a weighted addition, inaccordance with differences in gain signals, for the gain signal of again map having a large image size, and a gain signal for a gain maphaving a small image size.

In step S207, the gain processing unit 108 performs processing forapplying gain to the input image by using the composed gain map that wasgenerated in step S206. Letting the signal value of the input image atcoordinates (x, y) be in (x, y) and the value of the second gain map beGain(x, y), the signal value out(x, y) for the output signal after gainprocessing is represented as in Equation 2.

out(x, y)=Gain(x, y)×in(x, y)   (Equation 2)

When the processing by the gain processing unit 108 ends, the imageprocessing unit 1104 outputs the output image for which the gainprocessing has been performed (in other words tone conversion processinghas been performed), and ends the series of operations for thisprocessing.

In this way, in the gain processing in the present embodiment, gainprocessing to generate a final gain map is performed by composing thegain maps of each hierarchical level, with the input image as an upperhierarchical level and a reduced image as a lower hierarchical level. Insuch a case, configuration is such that, in a lower hierarchical level(namely an image having a relatively lower resolution), the gain map isgenerated by applying tone characteristics that are different for eachsubject region, based on the subject discrimination result. By applyingtone characteristics for each subject region in a lower hierarchicallevel in this way, it is possible to smooth (in other words reduce) aninfluence due to an incorrect detection of a subject discriminationresult near a boundary of a subject region.

Next, with reference to FIG. 4 explanation is given in detail regardingthe first tone characteristic and the second tone characteristic thatare used in step S203 and in step S204.

FIG. 4 illustrates a gain table in which an abscissa is an inputluminance signal and an ordinate is a gain signal, and the first tonecharacteristic is indicated as a tone characteristic 401 and the secondtone characteristic is indicated as a tone characteristic 402. Eachcharacteristic is such that gain is strongly applied at low luminances,and the gain weakens as the luminance becomes a high luminance. An imageillustrated on the right side of FIG. 4 illustrates an example of aninput image in the present embodiment, and includes a person region 411(corresponds to the person region 501), and a region other than that(corresponds to the region 502).

As explained with reference to FIGS. 5A to 5B, in a conventional method,there is a problem in that, when gain is applied to brighten the personregion 501, the region 502 that is not a person region also becomesbrighter, and the contrast of the region that is not a person regiondecreases. In contrast to this, in the present embodiment, in a range503 where brightnesses are overlapping, differing gain characteristicsare applied to a person region and a region other than this. In otherwords, in a luminance range illustrated by reference numeral 403 (FIG.4) as a range that corresponds to a person region, gains that greatlydiffer for each region are applied. For example, the second tonecharacteristic 402 has a characteristic of strongly applying gain sothat, for a person region, a face approaches an appropriate brightness.In contrast to this, the first tone characteristic 401 has acharacteristic such that, for a region that is not a person region, gainis made lesser than for a person region so that contrast does notdecrease by making a dark portion too bright. Furthermore, the firsttone characteristic 401 and the second tone characteristic 402 graduallyapproach the same characteristic in the range of luminances illustratedby reference numeral 404. By using tone characteristics such as these,it is possible to perform tone control so that the same or similarcharacteristics are achieved outside of a luminance range for whichseparating gains is desired. In addition, even in a case where erroneousextraction of a region discrimination result on a high-luminance sidehas occurred, it is possible to expect an improvement of robustness suchas a harmful effect due to an incorrect detection for regiondiscrimination not standing out, because there is mostly no differencein characteristics.

Note that, in the present embodiment, explanation was given for anexample of a case of using a person region as a subject region todiscriminate, but there is no limitation to a person region, and it ispossible to use another subject region that is meaningful, such as a skyregion or a vegetation region.

In the present embodiment as explained above, configuration is such thata final gain map is generated by composing gain maps with respected tohierarchized images. In such a case, configuration is such that, in areduced lower hierarchical level (namely an image having a relativelylower resolution), the gain map is generated by applying a plurality ofdifferent tone characteristics that are associated with each subjectregion. With such a configuration, in a case of applying tonecharacteristics that differ for each subject region, it is possible toenhance robustness for a region discrimination result for a subjectregion, and it is possible to perform tone processing having a higheffect in improving contrast. In other words, it is possible to, even ifthere is an error in discrimination of a subject region in tonecorrection that applies tone characteristics that differ for eachsubject region, reduce the influence of the error.

Furthermore, configuration is such that, a difference of gains for thetone characteristic for each region is made large in a range ofbrightnesses common for a desired subject region and a region other thanthat, and a difference of gains is made small in a range where thebrightness is not common. With such a configuration, it is possible tosatisfactorily maintain contrast in a region that is not a desiredsubject region. In addition, configuration is such that, the differenceof gain for each region is magnified in a range of luminances (lowluminance) where the difference in gain should be emphasized, and thedifference in gain is made small in a region of other luminances (highluminances). With such a configuration, it is possible to perform toneprocessing by subject region that increases robustness even with respectto region discrimination in a high-luminance region.

Second Embodiment

Next, explanation is given for a second embodiment. The secondembodiment differs from the first embodiment in that it uses the resultsof performing two or more types of subject determination processes tochange tone characteristics to be applied to separate regions, but theconfiguration of the digital camera 100 is the same. Accordingly, thesame reference numerals are given to the same elements, and overlappingexplanation is omitted, and predominantly explanation is given fordifferences.

FIG. 6 illustrates a configuration of the image processing unit 1104according to the present embodiment. In the present embodiment, thereare a first subject region discrimination unit 603 and a second subjectregion discrimination unit 604 for performing two differing types ofregion discrimination processing. In addition, there are three gainconversion units: the first gain conversion unit 104, a second gainconversion unit 601, and a third gain conversion unit 602. Furthermore,a by-region gain map composing unit 605 differs to the first embodiment.Details of each configuration are described later.

Next, explanation is given regarding gain processing according to thepresent embodiment, with reference to FIG. 7.

In step S701, the first subject region discrimination unit 603discriminates a meaningful first subject region from the input image.For example, a person region is discriminated as a first subject region.In a method of discriminating a subject region, it is possible to use apublicly known method that uses learning data or a feature amount of animage, similarly to in the first embodiment. For a subjectdiscrimination result, a signal value of 255 to 0 is output as a valueof 100 to 0% for a reliability, similarly to in the first embodiment.

In step S702, the second subject region discrimination unit 604discriminates a meaningful second subject region from the input image.For example, a sky region is discriminated as the second subject region.For a method of discrimination and a discrimination result, it issimilar to that in step S701 which is described above. In step S703, thefirst reduced image generating unit 101 and the second reduced imagegenerating unit 102 generate a first reduced image and a second reducedimage by performing reduction processing on the input image, similarlyto in the first embodiment.

In step S704, the first gain conversion unit 104 generates a gain mapfor each hierarchical level by applying the first tone characteristic tothe input image and the first reduced image and the second reduced imagethat are generated in step S703. The first tone characteristic is a tonecharacteristic that assumes application to a background region. Detailsof the first tone characteristic are described later.

In step S705, the second gain conversion unit 601 generates a gain mapthat corresponds to the second tone characteristic by applying thesecond tone characteristic to the second reduced image. The second tonecharacteristic is a tone characteristic that assumes application to aperson region. Details of the second tone characteristic are describedlater.

In step S706, the third gain conversion unit 602 generates a gain mapthat corresponds to the third tone characteristic by applying the thirdtone characteristic to the second reduced image. The third tonecharacteristic is a tone characteristic that assumes application to asky region. The third tone characteristic is described later.

In step S707, the by-region gain map composing unit 605 composes each ofthe gain maps that are generated in accordance with the first tonecharacteristic, the second tone characteristic, and the third tonecharacteristic and are based on the second reduced image. Forcomposition coefficients that are used in the composing, the firstregion discrimination result obtained in step S701 and the second regiondiscrimination result obtained in step S702 are used. Here, let thevalue of each of the gain maps respectively generated in accordance withthe first tone characteristic, the second tone characteristic, and thethird tone characteristic and corresponding to coordinates (x, y) of thesecond reduced image be Gain1(x, y), Gain2 (x, y), and Gain3 (x, y). Inaddition, let the signal value for discrimination of the first andsecond regions be respectively area1(x, y) and area2(x, y), and let themaximum value that can be taken for the signal value for discriminationof the first and second regions be area_MAX. Here, the by-region gainmap composing unit 605 can calculate Gaincomp(x, y), the value of thegain map resulting from composing, in accordance with Equation 3.Gaincomp_dash(x, y) is a partway result of composition.

$\begin{matrix}{{{Gaincomp\_ dash}\left( {x,y} \right)} = \frac{\begin{matrix}{{{\left( {{area\_ MAX} - {{area}\; 1\left( {x,y} \right)}} \right) \cdot {Gain}}\; 1\left( {x,y} \right)} +} \\{{area}\; 1{\left( {x,y} \right) \cdot {Gain}}\; 2\left( {x,y} \right)}\end{matrix}}{area\_ MAX}} & \left( {{Equation}\mspace{14mu} 3} \right) \\{{{Gaincomp}\left( {x,y} \right)} = \frac{\begin{matrix}{{{\left( {{area\_ MAX} - {{area}\; 2\left( {x,y} \right)}} \right) \cdot {Gaincomp\_ dash}}\left( {x,y} \right)} +} \\{{area}\; 2{\left( {x,y} \right) \cdot {Gain}}\; 2\left( {x,y} \right)}\end{matrix}}{area\_ MAX}} & \;\end{matrix}$

Note that, in the present embodiment, the region discrimination resultsarea1(x, y) and area2(x, y) are used unchanged, but a signal value afterapplying a low-pass filter, a bilateral filter, or the like to a regiondiscrimination result may be used.

In step S708, the hierarchical gain map composing unit 107 generates afinal gain map by composing the gain maps generated from each of theinput image and the first reduced image in step S704, and the gain mapgenerated based on the second reduced image in step S707. Explanation ofcomposition method for a gain map is omitted because it is similar tothat in the first embodiment.

In step S709, the gain processing unit 108 uses the gain map generatedin step S708 to perform gain processing with respect to the input image,similarly to in the first embodiment. When the gain processing by thegain processing unit 108 completes, the image processing unit 1104outputs the output image, and this series of operations ends.

Additionally, with reference to FIG. 8, explanation is given regardingthe first tone characteristic, the second tone characteristic, and thethird tone characteristic which are respectively used in theaforementioned step S704, step S705, and step S706. In a gain tablewhere an abscissa indicates an input luminance signal and an ordinateindicates a gain signal, the first tone characteristic corresponds to atone characteristic 801, the second tone characteristic corresponds to atone characteristic 802, and the third tone characteristic correspondsto a tone characteristic 803. Each characteristic is such that gain isstrongly applied at low luminances, and the gain weakens as theluminance becomes a high luminance. An image illustrated on the rightside of FIG. 8 illustrates an example of an input image in the presentembodiment, and includes a person region 811, a sky region 812, and aregion 813 that is apart from these.

As illustrated by FIGS. 9A to 9B, in the present embodiment,consideration is given to separating gain in a range 902 wherebrightnesses overlap for a sky region 812 and a region 813, in additionto a range 901 of brightnesses where a distribution of brightnessesoverlap for a person region 811 and a region 813 (that is not a personor a sky region). While the second tone characteristic differentiatesthe gain greatly with respect to the first tone characteristic 801 inthe luminance range 804, the third tone characteristic differentiatesthe gain greatly with respect to the second tone characteristic 802 inthe luminance range 807. In other words, with respect to the second tonecharacteristic 802 which strongly applies gain so that a face becomesbright for a person region, the first tone characteristic 801 is acharacteristic for making the gain weaker than for a person region, sothat contrast does not decrease in accordance with a dark portion thatis not a person region becoming too bright. In addition, with respect tothe third tone characteristic 803 which lessens gain so that gain is notapplied too much to a sky region, the first tone characteristic 801 hasa characteristic of smoothly applying gain to suppress a decrease incontrast due to tone compression from a medium luminance to a highluminance.

Furthermore, in the present embodiment, in addition to the second tonecharacteristic gradually approaching the same characteristic as thefirst tone characteristic in the luminance range 805, the third tonecharacteristic also gradually approaches the same characteristic as thefirst tone characteristic in the luminance range 806. With such aconfiguration, it is possible to perform tone control such that the samecharacteristic is achieved outside of luminance ranges where separationof gain is desired, similarly to in the first embodiment. Similarly toin the first embodiment, even in a case where erroneous extraction of aregion discrimination result on a high-luminance side occurs, it ispossible to expect an improvement of robustness such as a harmful effectnot standing out, because there is mostly no difference in tonecharacteristics.

In the present embodiment as explained above, in processing forgenerating a final gain map by composing gain maps that correspond tohierarchized images, configuration is such that the gain map isgenerated by applying a plurality of different tone characteristics thatare associated with each subject in a reduced lower hierarchical level.In particular, in the reduced lower hierarchical level, configuration issuch that the gain maps obtained from the plurality of different tonecharacteristics that are associated with each subject are composed basedon a plurality of region discrimination results. With such aconfiguration, even in a case of using region discrimination results fora plurality of subjects, it is possible to perform a tone correctionthat reduces an influence of each region discrimination result.

Note that, in the first embodiment and the second embodiment,explanation was given of an example of generating a gain map in whichtone characteristics are differentiated for each subject region in thesecond reduced image—in other words an image of a lowest level, butsimilar processing may be applied to the first reduced image.

In addition, configuration may be determined, in accordance with anoutput accuracy for the region discrimination processing (an outputresolution of a subject discrimination result or a reliability of theregion discrimination result), at which hierarchical level of an image(what degree of reduction) to apply the aforementioned processing forgenerating a gain map in which tone characteristics are differentiatedfor each subject region.

For example, FIGS. 10A and 10B illustrate tables for determining to whathierarchical level to apply the aforementioned processing for applyingthe differing tone characteristics in a case where a full size image isa first hierarchical level and an image of a lowest level having thesmallest image size is an Nth hierarchical level. The table illustratedin FIG. 10A illustrates a table that allows for selection of ahierarchical level in accordance with a reliability of a regiondiscrimination result. In contrast, the table illustrated in FIG. 10Bindicates a table that allows for selection of a hierarchical level inaccordance with an output resolution of a region discrimination result.

Note that, although in this example the reliability is something thatindicates a maximum value out of reliabilities outputted by a regiondiscrimination result, for example an average value of a discriminationresult may be used. In the table of FIG. 10A, the gain map is generatedby causing the tone characteristic per subject region to graduallydiffer in hierarchical levels above the Nth hierarchical level, as thereliability of the region discrimination result approaches 100%. Byconfiguring as in FIG. 10A, it is possible to control, in accordancewith a reliability of a region discrimination result, how much toreflect an effect due to tone processing by subject region in a gain mapused in final gain processing. In addition, as in FIG. 10B, it is alsopossible to control, in accordance with what level the size of an outputresolution for region discrimination is, whether to perform processingto generate a gain map by causing the tone characteristic for a subjectregion to differ from the Nth hierarchical level to the hierarchicallevel of that size.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2016-206092, filed Oct. 20, 2016, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: animage generation unit configured to generate one or more reduced imagesthat are reduced in a stepwise fashion based on an input image, theinput image and the reduced images forming hierarchized images; a gainmap generation unit configured to generate a gain map for eachhierarchical level that represents a gain at each pixel position of animage, by applying a first tone characteristic to each of thehierarchized images and also further applying a second tonecharacteristic to at least one of the one or more reduced images; and acomposing unit configured to compose gain maps generated for eachhierarchical level to generate a composed gain map that represents gainapplied to each pixel position of the input image, wherein the gain mapgeneration unit applies, for each subject region, the first tonecharacteristic and the second tone characteristic to the at least one ofthe one or more reduced images, and wherein the first tonecharacteristic and the second tone characteristic are mutually differenttone characteristics associated with a respective subject regionincluded in the input image.
 2. The apparatus according to claim 1,further comprising: a discrimination unit configured to discriminate apredetermined subject region from the input image, wherein the gain mapgeneration unit, based on a discrimination result by the discriminationunit, applies, for each subject region, the first tone characteristicand the second tone characteristic to the at least one of the one ormore reduced images.
 3. The apparatus according to claim 2, wherein thesecond tone characteristic is a tone characteristic that is associatedwith the predetermined subject region included in the input image, andthe first tone characteristic is a tone characteristic that isassociated with a region that is different to the predetermined subjectregion.
 4. The apparatus according to claim 2, wherein the gain mapgeneration unit differentiates a number of the one or more reducedimages to which the first tone characteristic and the second tonecharacteristic are applied in accordance with an accuracy of regiondiscrimination by the discrimination unit.
 5. The apparatus according toclaim 2, wherein the gain map generation unit applies the first tonecharacteristic and the second tone characteristic to the one or morereduced images for which a degree of reduction decreases as accuracy ofregion discrimination by the discrimination unit increases.
 6. Theapparatus according to claim 2, wherein, for the first tonecharacteristic and the second tone characteristic, a difference in gainapplied to a range of luminance signals that are not in common with thepredetermined subject region and a region that is not the predeterminedsubject region is smaller than a difference in gain applied to a rangeof luminance signals that are in common with the predetermined subjectregion and the region that is not the predetermined subject region. 7.The apparatus according to claim 1, wherein the gain map generation unitincludes a most reduced image in the at least one of the one or morereduced images.
 8. The apparatus according to claim 2, wherein thediscrimination unit discriminates from the input image a secondpredetermined subject region that differs from the predetermined subjectregion, and wherein the gain map generation unit, based on thediscrimination result by the discrimination unit, further applies athird tone characteristic that is different to the first tonecharacteristic and the second tone characteristic and is associated withthe second predetermined subject region to the at least one of the oneor more reduced images.
 9. An image processing apparatus, comprising: animage reduction unit configured to generate one or more reduced imagesby reducing an input image and generate an image group that includes theinput image and the one or more reduced images; and a gain mapgeneration unit configured to respectively generate a gain map thatindicates a gain value at each position in an image by applying apredetermined tone characteristic to each image in the image group,wherein the gain map generation unit generates the gain map by applying,to the image out of the image group for which a resolution is relativelylowest, tone characteristics for which a characteristic differs for eachsubject region included in the image; a gain map composing unitconfigured to generate a composed gain map by composing a plurality ofgain maps generated based on each image of the image group; and an imageconversion unit configured to perform a tone conversion by using thecomposed gain map on the input image.
 10. The apparatus according toclaim 9, further comprising: a discrimination unit configured todiscriminate a predetermined subject region from the image, wherein thegain map generation unit generates the gain map by applying the tonecharacteristics for which the characteristic differs for each subjectregion included in the image, based on a discrimination result by thediscrimination unit.
 11. The apparatus according to claim 10, whereinthe gain map generation unit differentiates a number of reduced imagesto which different tone characteristics are applied for each subjectregion included in an image in accordance with an accuracy of regiondiscrimination by the discrimination unit.
 12. The apparatus accordingto claim 10, wherein the gain map generation unit increases a number ofreduced images to which different tone characteristics are applied foreach subject region included in an image as an accuracy of regiondiscrimination by the discrimination unit increases.
 13. A controlmethod of an image processing apparatus, the method comprising:generating one or more reduced images that are reduced in a stepwisefashion based on an input image, the input image and the reduced imagesforming hierarchized images; generating a gain map for each hierarchicallevel that represents a gain at each pixel position of an image, byapplying a first tone characteristic to each of the hierarchized imagesand also further applying a second tone characteristic to at least oneof the one or more reduced images; and composing gain maps generated foreach hierarchical level to generate a composed gain map that representsgain applied to each pixel position of the input image, wherein, foreach subject region, the first tone characteristic and the second tonecharacteristic are applied to the at least one of the one or morereduced images, and wherein the first tone characteristic and the secondtone characteristic are mutually different tone characteristicsassociated with a respective subject region included in the input image.14. A control method of an image processing apparatus, the methodcomprising: generating at least one or more reduced images by reducingan input image and generate an image group that includes the input imageand the at least one or more reduced images; respectively generating again map that indicates a gain value at each position in an image byapplying a predetermined a tone characteristic to each image in theimage group, wherein the gain map is generated by applying, to an imageout of the image group for which a resolution is relatively low, tonecharacteristics for which a characteristic differs for each subjectregion included in the image; generating a composed gain map bycomposing a plurality of gain maps generated based on each image of theimage group; and performing a tone conversion by using the composed gainmap on the input image.
 15. A non-transitory computer-readable storagemedium storing a computer program for causing a computer to function asan image processing apparatus comprising: an image generation unitconfigured to generate one or more reduced images that are reduced in astepwise fashion based on an input image, the input image and thereduced images forming hierarchized images; a gain map generation unitconfigured to generate a gain map for each hierarchical level thatrepresents a gain at each pixel position of an image, by applying afirst tone characteristic to each of the hierarchized images and alsofurther applying a second tone characteristic to at least one of the oneor more reduced images; and a composing unit configured to compose gainmaps generated for each hierarchical level to generate a composed gainmap that represents gain applied to each pixel position of the inputimage, wherein the gain map generation unit applies, for each subjectregion, the first tone characteristic and the second tone characteristicto the at least one of the one or more reduced images, and wherein thefirst tone characteristic and the second tone characteristic aremutually different tone characteristics associated with a respectivesubject region included in the input image.